Imaging display device, wearable device, and imaging display system

ABSTRACT

An imaging unit includes a plurality of photoelectric conversion elements, a processing unit, and a display unit. The processing unit processes a signal transmitted from the imaging unit. The display unit displays an image based on the signal transmitted from the processing unit. The imaging unit acquires first image information at a first time. The processing unit generates first prediction image information at a second time later than the first time based on the first image information. Moreover, the display unit displays an image based on the first prediction image information.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 17/336,056, filed on Jun. 1, 2021, which is a continuation ofU.S. patent application Ser. No. 16/562,304, filed on Sep. 5, 2019 andissued as U.S. Pat. No. 11,048,465 on Jun. 29, 2021, which claimspriority from Japanese Patent Application No. 2018-174104, filed Sep.18, 2018, and Japanese Patent Application No. 2019-126318, filed Jul. 5,2019, which are hereby incorporated by reference herein in theirentireties.

BACKGROUND Field

The present disclosure relates to an imaging display device, a wearabledevice, and an imaging display system.

Description of the Related Art

A wearable device called a head-mounted display or a pair ofsmart-glasses having an imaging display device has been known. In onesystem of the above-described wearable device, a scenery in front of auser is captured as an image through the imaging display device, and thecaptured image is displayed on a display device. In the above-describedsystem, the user can feel as if the user was directly watching theexternal scenery even though watching it via the display apparatus.

In order to miniaturize the above-described display apparatus, JapanesePatent Application Laid-Open No. 2002-176162 discusses a technique forarranging a photodiode and an electroluminescence (hereinafter, “EL”)element on a same substrate in a matrix state.

SUMMARY

According to an aspect of the present disclosure, an imaging displaydevice includes an imaging unit including a plurality of photoelectricconversion elements, a processing unit configured to process a signaltransmitted from the imaging unit, and a display unit configured todisplay an image based on the signal transmitted from the processingunit, wherein the imaging unit acquires first image information at afirst time, wherein the processing unit generates first prediction imageinformation at a second time later than the first time based on thefirst image information, and wherein the display unit displays an imagebased on the first prediction image information.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic diagram illustrating an imaging display deviceaccording to a first exemplary embodiment. FIG. 1B is a schematicdiagram illustrating a variation example of the imaging display deviceaccording to the first exemplary embodiment. FIG. 1C is a schematicdiagram illustrating another variation example of the imaging displaydevice according to the first exemplary embodiment. FIG. 1D is aschematic diagram illustrating still another example of the imagingdisplay device according to the first exemplary embodiment.

FIGS. 2A and 2B are diagrams illustrating operation of the imagingdisplay device according to the first exemplary embodiment.

FIG. 3 is a diagram illustrating a comparison example of the operationof the imaging display device according to the first exemplaryembodiment.

FIG. 4 is a schematic diagram illustrating an imaging display deviceaccording to a second exemplary embodiment.

FIG. 5 is a diagram illustrating operation of the imaging display deviceaccording to the second exemplary embodiment.

FIGS. 6A and 6B are diagrams illustrating operation of an imagingdisplay device according to a third exemplary embodiment.

FIG. 7 is a schematic diagram illustrating an imaging display deviceaccording to a fourth exemplary embodiment.

FIG. 8 is a diagram illustrating operation of an imaging display deviceaccording to a fifth exemplary embodiment.

FIG. 9 is a diagram illustrating operation of an imaging display deviceaccording to a sixth exemplary embodiment.

FIGS. 10A and 10B are schematic diagrams illustrating a wearable device.FIG. 10C is a cross-sectional diagram schematically illustrating apositional relationship between an imaging unit and a display unit.FIGS. 10D and 10E are planar diagrams schematically illustrating apositional relationship between the imaging unit and the display unit.

FIG. 11 is a schematic diagram illustrating an imaging display system.

FIG. 12 is a schematic diagram illustrating an imaging display deviceaccording to a tenth exemplary embodiment.

FIG. 13 is a schematic diagram illustrating operation of an imagingdisplay device according to an eleventh exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, exemplary embodiments will be described with reference tothe appended drawings. In below-described exemplary embodiments,description of a configuration similar to that already described inanother exemplary embodiment will be omitted. Further, the exemplaryembodiments can be changed or combined as appropriate.

A first exemplary embodiment will be described below. The presentexemplary embodiment will be described with reference to FIGS. 1A to 1Dand FIGS. 2A and 2B. FIG. 1A is a schematic diagram illustrating animaging display device 100 according to the present exemplaryembodiment. The imaging display device 100 includes an imaging unit 101,a processing unit 102, and a display unit 103.

The imaging unit 101 includes a plurality of light receiving elements.For example, photoelectric conversion elements are used as the lightreceiving elements. The light receiving elements execute imagingoperation for acquiring image information by converting light enteringfrom the outside (external information) to electric signals. Based onthe image information from the imaging unit 101, the processing unit 102generates information about an image that is to be captured by theimaging unit 101 in the future (hereinafter, referred to as “predictionimage information”). The display unit 103 includes a plurality of lightemitting elements. Each of the plurality of light emitting elementsconverts an electric signal to light. The display unit 103 displays(outputs) an image based on the prediction image information generatedby the processing unit 102. A plurality of pixels is arranged on theimaging unit 101 and the display unit 103 in an array state. Each of thepixels arranged on the imaging unit 101 includes at least one lightreceiving element, and each of the pixels arranged on the display unit103 includes at least one light emitting element. The processing unit102 receives image information from the imaging unit 101 and outputsprediction image information to the display unit 103. Further, theprocessing unit 102 can output a control signal for executing imagingoperation and a control signal for executing display operation to theimaging unit 101 and the display unit 103, respectively.

Herein, a variation example of the imaging display device 100 accordingto the present exemplary embodiment illustrated in FIG. 1A will bedescribed. FIG. 1B is a schematic diagram illustrating the variationexample of the imaging display device 100 according to the presentexemplary embodiment in FIG. 1A. In an imaging display device 120, theprocessing unit 102 illustrated in FIG. 1A includes an artificialintelligence (AI) unit 104 on which AI is mounted. In the presentexemplary embodiment, the AI unit 104 has a deep learning (deepstructured learning) function. In FIG. 1B, image information captured bythe imaging unit 101 is converted to prediction image information by theprocessing unit 102 including the AI unit 104.

FIG. 1C is a schematic diagram illustrating a variation example of theimaging display device 100 of the present exemplary embodimentillustrated in FIG. 1A. A processing unit 102 of an imaging displaydevice 130 communicates with a processing apparatus 105. The processingunit 102 and the processing apparatus 105 are connected to each othervia a network. The processing apparatus 105 is arranged on the outsideof the imaging display device 130, e.g., the cloud. In the imagingdisplay device 130, the AI unit 104 is included not in the processingunit 102 but in the processing apparatus 105. The processing unit 102and the processing apparatus 105 exchange information, and predictionimage information is thereby generated based on image information. InFIG. 1C, the image information captured by the imaging unit 101 isconverted to prediction image information by the processing unit 102that has acquired the information from the processing apparatus 105. Insuch a manner, the imaging display device 130 can generate predictionimage information using the information stored in the externalapparatus.

FIG. 1D is a schematic diagram illustrating a variation example of theimaging display device 100 of the present exemplary embodimentillustrated in FIG. 1A. A processing unit 102 of an imaging displaydevice 140 includes an AI unit 104. The processing unit 102 communicateswith a processing apparatus 106, and the processing apparatus 106further communicates with another processing apparatus 105. Theprocessing apparatus 106 is arranged on the cloud and stores data. Theprocessing apparatus 105 including the AI unit 104 is arrangedseparately from the imaging display device 140 and the processingapparatus 106. The processing unit 102 and the processing apparatus 106,and the processing apparatus 106 and the processing apparatus 105 arerespectively connected to each other via a network. In FIG. 1D, theprocessing unit 102 receives setting information stored in theprocessing apparatus 106 and generates prediction image informationbased on the setting information. The setting information includes basicinformation about an environment or a target object, and various valuesused for generating the prediction image information. Further, theprocessing unit 102 transmits a plurality of pieces of informationincluding the image information from the imaging unit 101 to theprocessing apparatus 106. The plurality of pieces of information istransmitted to the processing apparatus 105 via the processing apparatus106. The processing apparatus 105 generates the various values used forgenerating the prediction image information based on the plurality ofpieces of received information, and transmits the generated values tothe processing apparatus 106. The processing apparatus 106 updates thebasic information and values stored therein and retains the updatedinformation and the various values as new information. As describedabove, the imaging display device 140 can generate the prediction imageinformation using the information stored in the external apparatus.

In FIG. 1A, the processing unit 102 predicts image information that isto be captured by the imaging unit 101 in the future based on the imageinformation acquired by the imaging unit 101, and transmits the imageinformation to the display unit 103 as prediction image information.Further, the processing unit 102 can process the other types ofinformation such as temperature/humidity information, accelerationinformation, and pressure information together with the imageinformation. The same can be also said for the processing unit 102 inFIG. 1B, the processing unit 102 and the processing apparatus 105 inFIG. 1C, and the processing unit 102 and the processing apparatuses 105and 106 in FIG. 1D, which are described in the variation examples.

Subsequently, the operation of the imaging display device 100 of thepresent exemplary embodiment will be described with reference to FIGS.2A and 2B. FIGS. 2A and 2B are diagrams illustrating the operation ofthe imaging display device 100 of the present exemplary embodiment and arelationship between image information and prediction image informationwith respect to one frame at a certain time. In FIGS. 2A and 2B, imageinformation at time Tn is expressed as “An”, whereas future imageinformation (prediction image information) processed by the processingunit 102 is expressed as “Bn”.

The operation of the imaging display device 100 according to the presentexemplary embodiment will be described with reference to FIG. 2A. Inthis operation, the imaging unit 101 executes imaging operation foracquiring image information A⁻² at time T⁻², image information A⁻¹ attime T⁻¹, image information A₀ at time T₀, and image information A₊₁ attime T₊₁. Next, the processing unit 102 generates pieces of predictionimage information B₀, B₊₁, and B₊₂ based on the pieces of input imageinformation A⁻¹, A₀, and A₊₁, respectively. Then, the processing unit102 outputs the pieces of prediction image information B₀, B₊₁, and B₊₂to the display unit 103. The display unit 103 executes display operationfor displaying an image based on the prediction image information B₀ attime T₀, an image based on the prediction image information B₊₁ at timeT₊₁, and an image based on the prediction image information B₊₂ at timeT₊₂.

In other words, the imaging unit 101 executes imaging operation foracquiring the image information A⁻¹ at a certain time T⁻¹, and executesimaging operation for acquiring the image information A₀ different fromthe image information A⁻¹ at the time To later than the certain time Li.The display unit 103 executes display operation for displaying an imagebased on the prediction image information B₀ generated based on theimage information A⁻¹ at the time To. Further, at the time T₊₁ laterthan the time To, the imaging unit 101 executes imaging operation foracquiring the image information A₊₁ different from the image informationA₀. Then, the display unit 103 executes display operation for displayingan image according to the prediction image information B₊₁ generatedfrom the image information A₀.

Herein, a comparison example will be described with reference to FIG. 3. In an imaging display apparatus which does not have the processingunit 102 described in the present exemplary embodiment, the imageinformation A⁻¹ captured by the imaging unit 101 at the time T⁻¹ isdisplayed by the display unit 103 at the time T₀.

Difference between a configuration in which the prediction imageinformation of the present exemplary embodiment in FIG. 2A is displayedand a configuration in which the image information captured by theimaging unit 101 in FIG. 3 is displayed as it is will be described. Ifthe current time is the time To, the image information A₀ corresponds tothe actual phenomenon (i.e., real image) at that time. Information usedas a source of an image to be displayed by the display unit 103 of thepresent exemplary embodiment is the prediction image information B₀. Ifthe image information captured by the imaging unit 101 is used as it is,the information used as a source of the image displayed by the displayunit 103 is the image information A⁻¹. Herein, an amount of change ofthe image information is expressed as “(Image Information A⁻¹−ImageInformation A₀)≥(Prediction Image Information B₀−Image Information A₀)”.Accordingly, with the configuration described in the present exemplaryembodiment, it is possible to acquire an imaging display device thatexecutes reducing a difference between the actual phenomenon and thedisplayed image.

A timing for displaying the prediction image information of the presentexemplary embodiment will be described. The processing unit 102 of thepresent exemplary embodiment generates prediction image information toreduce the lag time between the image information captured by theimaging unit 101 at a certain time and the image to be displayed by thedisplay unit 103. It is desirable that a timing for displaying theprediction image information be set as follows.

First, it is assumed that an image is captured by the imaging unit 101at an optional time Tn. The processing unit 102 generates predictionimage information based on the image information acquired at the timeTn. The time when the display unit 103 displays an image based on theprediction image information generated with respect to the time Tn isexpressed as time Tm. Herein, a difference ΔT between the imaging timingand the display timing can be expressed by a formula 1.

ΔT=Tn−Tm  Formula 1

A display frame rate DFR (fps: frame per second) represents the numberof images displayed by the display unit 103 per second. The imagingdisplay device is controlled in such a manner that the difference ΔTsatisfies a formula 2. More preferably, the imaging display device iscontrolled in such a manner that the difference ΔT satisfies a formula3.

−2/DFR≤ΔT≤2/DFR  Formula 2

−1/DFR≤ΔT≤1/DFR  Formula 3

For example, when the display frame rate is 240 fps, time taken for oneimage (one frame) to be displayed after being captured is approximately4×10⁻³ sec. Accordingly, the difference ΔT is expressed as follows.

−4×10⁻³ ≤ΔT≤4×10⁻³  Formula 4

By displaying the image based on the prediction image information at theabove-described timing, it is possible to display a moving image with asmall amount of lag time between the real image and the display image.This moving image display can be called “real-time display”.Accordingly, in the present exemplary embodiment, real-time display or,in a precise sense, pseudo real-time display, can be executed. Thepresent disclosure is efficiently applied to a moving image in additionto a still image.

Further, the above-described timing difference can be also used forgenerating the prediction image information in addition to displayingthe image at the above-described timing. Image information captured bythe imaging unit 101 at an optional time is expressed as “An”. Imageinformation displayed by the display unit 103 at the same time isexpressed as “Dn”. Herein, a difference between the image information Anand the image information Dn, i.e., a temporal difference between theimage information An and the image information Dn, can be expressed asΔA=Dn−An. In the exemplary embodiment in FIG. 2A, the image informationDn is equal to the image information Bn (Dn=Bn). In other words, atemporal difference between the image information captured by theimaging unit 101 at a certain time, i.e., an actual phenomenon (realimage) at that time, and the image information displayed by the displayunit 103 may be ±4×10⁻³ seconds. When the temporal difference betweenthe pieces of image information is ±4×10⁻³ sec., an image displayed bythe display unit 103 is a delayed image that is delayed for 4×10⁻³ sec.with respect to a real image at a certain time, or a future image thatis 4×10⁻³ sec. after the real image at the certain time. It is desirablethat the prediction image information be generated under theabove-described condition. In addition, for example, the imageinformation An and the image information Dn can be compared by using rawdata of the image information An and the image information Dn. Then, theimage information Dn may fall within a range of ±4×10⁻³ sec. when aroot-mean-square of the difference is calculated. The processing unit102 sets various parameters for generating next prediction imageinformation by using this difference information.

The lag time occurs when the image information captured by the imagingunit 101 is displayed on the display unit 103 as illustrated in FIG. 3 .Particularly, the lag time becomes 100×10⁻³ sec. when additional imageprocessing is executed. However, by generating the prediction imageinformation of the present exemplary embodiment, an image without atemporal difference with the real image can be displayed.

Dark-field image processing for increasing the luminance of a darkimage, image enlargement processing for enlarging and displaying a smallobject, and thermal display processing for displaying a thermal imageare given as examples of the additional image processing. Through theprocessing according to the present exemplary embodiment, real-timedisplay can be executed even if additional time is necessary forexecuting the above-described image processing.

Next, operation illustrated in FIG. 2B will be described. In thisoperation, the imaging unit 101 executes imaging operation for acquiringimage information A⁻² at a time T⁻², image information A⁻¹ at a timeT⁻¹, image information A₀ at a time T₀, and image information A₊₁ at atime T₊₁. The processing unit 102 generates pieces of prediction imageinformation B₊₁, B₊₂, and B₊₃ based on the input image information A⁻¹,A₀, and A₊₁, respectively. Then, the processing unit 102 outputs theprediction image information B₊₁, B₊₂, and B₊₃ to the display unit 103.The display unit 103 executes display operation for displaying an imagebased on the prediction image information B₊₁ at the time To, an imagebased on the prediction image information B₊₂ at time the T₊₁, and animage based on the prediction image information B₊₃ at the time T₊₂. Inother words, the processing unit 102 predicts image information that isto be captured at the time To and the display unit 103 displays an imagebased on the predicted image information at the time To. In such amanner, information with respect to a future time later than the imagingtime can be displayed at the imaging time. By consecutively repeatingthe above-described operation, images at a future time later than thereal image are displayed consecutively, i.e., the image can be displayedas a video image.

Image information used as a source of prediction image information willbe described. For example, in FIG. 2A, the prediction image informationB₀ is generated based on the image information A⁻¹. In FIG. 2B, theprediction image information B₊₁ is generated based on the imageinformation A⁻¹. In other words, one piece of prediction imageinformation is generated based on one piece of image information.However, one piece of prediction image information may be generatedbased on two or more pieces of image information. For example, in FIG.2A, the prediction image information B₀ may be generated based on thepieces of image information A⁻² and A⁻¹, and in FIG. 2B, the predictionimage information B₊₁ may be generated based on the pieces of imageinformation A⁻² and A⁻¹. Accordingly, the prediction image informationcan be generated by using at least a piece of image information.

A frame rate of the present exemplary embodiment will be described.First, the number of pieces of image information acquired by the imagingunit 101 per second is defined as an imaging frame rate SFR (fps). Then,as described above, the number of pieces of image information displayedby the display unit 103 per second is defined as a display frame rateDFR (fps). At this time, a relationship between the frame rates in FIGS.2A and 2B in the present exemplary embodiment is expressed as “SFR=DFR”.However, the imaging frame rate and the display frame rate may bedifferent from each other. Specifically, it is desirable that the framerates satisfy a condition “SFR≥DFR” because prediction image informationcan be generated from a plurality of pieces of captured imageinformation.

Next, a configuration of the imaging display device 100 will bedescribed. Examples of a photoelectric conversion element included inthe imaging unit 101 includes a photodiode, a photogate, and aphotoelectric conversion film. For example, silicon, germanium, indium,gallium, and arsenic can be used as the materials of the photodiode andthe photogate. A positive-negative (P-N) junction type photodiode, apositive-intrinsic-negative (PIN) type photodiode, and an avalanche typephotodiode can be given as the examples of the photodiode.

For example, a complementary metal-oxide semiconductor (CMOS) imagesensor can be used as the imaging unit 101, and the CMOS image sensormay be a front-face illumination type or a back-face illumination type.Further, the CMOS image sensor may have a structure in which asemiconductor substrate having a photodiode arranged thereon and asemiconductor substrate having a scanning circuit and a control circuitarranged thereon are laminated on each other.

Further, a material of the photoelectric conversion film may be anorganic material or an inorganic material. The organic photoelectricconversion film has a structure having at least one organic layer forexecuting photoelectric conversion, which is arranged at a positionbetween a pair of electrodes. The organic photoelectric conversion filmmay also have a structure having a plurality of organic layers laminatedand arranged at a position between a pair of electrodes. The organiclayer may be made of a single material or a plurality of materials mixedtogether. Further, the organic layer can be formed by vacuum vapordeposition processing or coating processing. For example, a quantum-dottype photoelectric conversion film using a quantum-dot thin-film layercontaining fine semiconductor crystals instead of an organic layer, or aperovskite-type photoelectric conversion film including a photoelectricconversion layer consisting of transition metal oxide having aperovskite structure is used as the inorganic photoelectric conversionfilm.

The display unit 103 includes a plurality of light emitting elements.The light emitting element may be a liquid crystal display (LCD), aninorganic light emitting diode (LED), an organic LED (OLED), or aquantum dot LED (QLED). For example, materials such as aluminum,gallium, arsenic, phosphorus, indium, nitrogen, selenium, zinc, diamond,zinc oxide, and/or a perovskite semiconductor are used for the inorganicLED. The inorganic LED having the P-N junction structure formed by usingthe above-described materials emits light having energy (wavelength)corresponding to a bandgap between the above-described materials. Forexample, the organic LED includes a light emitting layer containing atleast one organic light emitting material arranged at a position betweena pair of electrodes. The organic LED may include a plurality of lightemitting layers, and may have a structure having a plurality of organiclayers laminated one on top of the other. The light emitting layer maybe made of a single material or a plurality of materials mixed together.Light emitted from the light emitting layer may be fluorescent light orphosphorescence light, and the light may be monochromatic light (e.g.,blue, green, or red light) or white light. Further, the organic layercan be formed by vacuum vapor deposition processing or coatingprocessing.

Further, the imaging display device may have a structure in which atleast three chips of the imaging unit 101, the processing unit 102, andthe display unit 103 are laminated and electrically connected to eachother by semiconductor processing.

In a case where the imaging display device 100 of the present exemplaryembodiment is used as a wearable device, it is desirable that an amountof data to be processed by the processing unit 102 be as small aspossible. This is because the wearable device needs to be reduced insize, weight, and thickness as much as possible, and a chip size of theprocessing unit 102 can be reduced further if a data processing load issmaller. In order to reduce the data processing load, the AI processingmay be executed by another apparatus (e.g., one provided on the cloud)as illustrated in FIG. 1C or 1D. Further, a method for lowering theresolution of a portion other than the line-of-sight area, a method forprocessing a portion other than the line-of-sight area to be a stillimage, or a method for processing a portion other than the line-of-sightarea in monochrome instead of color can be used as a method for reducinga processing amount.

If it takes long time to capture and display an image of an actualphenomenon such as scenery, there arises a difference between the actualphenomenon and the displayed image. For example, if there is adifference between the actual phenomenon and the display image, the usercannot perform operation for capturing a moving object. However,according to the present exemplary embodiment, it is possible to providean imaging display device that executes reducing a temporal differencebetween the actual phenomenon and the display image. Therefore, it ispossible to perform the operation for capturing a moving object.

Hereinafter, a second exemplary embodiment will be described. Thepresent exemplary embodiment will be described with reference to FIGS. 4and 5 . FIG. 4 is a schematic diagram illustrating an imaging displaydevice 400 according to the present exemplary embodiment. Similar to theimaging display device 130 illustrated in FIG. 1C, the imaging displaydevice 400 includes an imaging unit 101, a processing unit 102, adisplay unit 103, and a processing apparatus 105. The imaging displaydevice 400 further includes a detection unit 107 for detectingenvironmental information. In the imaging display device 400, each ofthe processing unit 102 and the processing apparatus 105 includes an AIunit 104.

The detection unit 107 includes at least one sensor. The sensor candetect at least a piece of environmental information. Information aboutatmospheric temperature, water temperature, humidity, atmosphericpressure, water pressure, and luminance are given as examples of theenvironmental information. Further, the detection unit 107 may alsoacquire physical information such as an acceleration rate of an objectcaptured by the imaging unit 101. In the present exemplary embodiment,although the detection unit 107 is built into the imaging display device400, the detection unit 107 may be externally arranged.

For example, prediction image information is generated by the AI unit104 having a deep learning function based on the image informationacquired from the imaging unit 101 and the environmental informationacquired from the detection unit 107. At this time, prediction imageinformation with respect to a time in which time taken for theprocessing unit 102 to execute processing is taken into consideration isgenerated. In other words, if an image is captured at a certain time,prediction image information with respect to a future time is generatedwhile adding time necessary for executing processing to the certain timein addition to time taken for capturing and displaying the image.Details of the operation are similar to the details described in thefirst exemplary embodiment.

Subsequently, processing executed by the AI unit 104 will be describedusing a scene where persons are playing baseball as an example. FIG. 5is a diagram illustrating the operation of the imaging display device ofthe present exemplary embodiment. FIG. 5 illustrates four types ofexamples of images captured at four different times T⁻³, T⁻², T⁻¹, andT₀.

A real image (in the light) in FIG. 5 illustrates a real image. Herein,the real image refers to an image captured by the imaging unit 101 ateach of times T⁻³, T⁻², T⁻¹, and T₀. The real image (in the light)illustrates a real image captured at each of the times T⁻³, T⁻², T⁻¹,and T₀ when persons are playing baseball in a bright environment (in thelight). Because a pitcher and a ball can be recognized clearly, a battercan hit and return the ball at the time To.

A real image (in the dark) illustrates a real image. Herein, the realimage refers to the image captured by the imaging unit 101 at each oftimes T⁻³, T⁻², T⁻¹, and T₀. The real image (in the dark) illustrates areal image captured at each of the times T⁻³, T⁻², T⁻¹, and T₀ when thepersons are playing baseball in a dark environment (in the dark). Sincea ball cannot be seen by the naked eyes in such a dark environment, thebatter cannot hit and return the ball at the time To. Therefore,although movement and positions of the pitcher and the ball in the realimages in the light and the real images in the dark are the same at therespective times T⁻³, T⁻², and T⁻¹, different results are acquired atthe time T₀.

The images of a comparison example illustrate a case in which theimaging display device of the comparison example is used when personsare playing baseball in the dark will be described. The images of thecomparison example represent images acquired by additionally executingimage processing on the real images captured in the dark and displayedon the imaging display device. Even in a state illustrated in the realimages captured in the dark, where the object cannot be seen by thenaked eyes, images just like the real images captured in the light canbe displayed by executing additional processing on the captured imageinformation. The additional image processing is processing forincreasing the luminance of an image captured in the dark. By executingthe above processing, the ball is visible to the batter. Because of thelag time, however, a position of the ball at the time Li is differentfrom the position in the real image. Therefore, the batter cannot hitand return the ball at the time To.

In the images of the present exemplary embodiment, an exemplaryembodiment in which the imaging display device of the present exemplaryembodiment is used when persons are playing baseball in the dark will bedescribed. The images of the present exemplary embodiment are imagesdisplayed on the imaging display device 100 after additional imageprocessing is executed on the real images captured in the dark. Even ina state illustrated in the real images captured in the dark, where theobject cannot be seen by the naked eyes, images just like the realimages captured in the light can be displayed by executing additionalprocessing on the captured image information. Further, images using theprediction image information which do not have lag time, as described inthe imaging display device 100 of the present exemplary embodiment, canbe displayed. Accordingly, the batter can hit and return the ball at thetime To as if the batter is playing baseball in the light. At the timesT⁻³, T⁻², T⁻¹, and T₀, movement and positions of the pitcher and theball in the images of the present exemplary embodiment and those in thereal image captured in the light are substantially the same.

As in the present exemplary embodiment, images can be displayed in realtime by the imaging display device using the prediction imageinformation. Thus, a position of a moving object such as a ball can berecognized precisely. Further, by use of the prediction imageinformation, images can be displayed in real time even if the additionalprocessing is executed on the captured image information. The imagingdisplay device according to the present exemplary embodiment isdesirable for capturing and displaying movement of a moving object insports such as baseball.

Further, in the example illustrated in FIG. 5 , the detection unit 107detects information about a wind direction and a wind speed and outputsthe detected information to the processing unit 102. Based on the aboveinformation to predict a speed or a course of the ball, the processingunit 102 generates the prediction image information on the ball.Further, the detection unit 107 can determine whether the additionalimage processing is to be executed by the processing unit 102 bydetecting luminance information.

Further, in the imaging display device 400 of the present exemplaryembodiment, it is desirable that the imaging frame rate SFR be greaterthan the display frame rate DFR. For example, the imaging frame rate SFRand the display frame rate DFR may be set to 500 fps and 60 fps,respectively. Because the imaging frame rate SFR is greater than thedisplay frame rate DFR (SFR>DFR), one piece of prediction imageinformation can be generated based on a number of pieces of imageinformation. Therefore, an agreement rate between the display image andthe real image is increased, so that movement of the moving object canbe displayed precisely.

Further, various types of information transmitted to the externalprocessing apparatus 105 in FIG. 1D include various types of informationacquired by the detection unit 107 of the present exemplary embodimentillustrated in FIG. 4 . For example, pitching data of a battery, data ona pitching form of a pitcher, weather information, and user (batter)information are basic information stored in the processing apparatus106.

Hereinafter, a third exemplary embodiment will be described. In thepresent exemplary embodiment, another operation of the imaging displaydevice 400 described in the second exemplary embodiment will bedescribed. FIGS. 6A and 6B are diagrams illustrating the operation of animaging display device of the present exemplary embodiment. FIG. 6Aillustrates a real image, whereas FIG. 6B illustrates an image based onprediction image information.

The real image illustrated in FIG. 6A can be referred to as imageinformation 600 acquired by the imaging unit 101 at a certain time. Theprocessing unit 102 detects a moving object 602 by using imageinformation captured before the certain time and at least two pieces ofimage information included in the image information 600. Characteristicstill objects 601 in the scenery may be detected simultaneously. Thecharacteristic still objects 601 can be identified by line-of-sightdetection described below. The line-of-sight detection refers to an eyetracking. In FIG. 6A, for example, two trees are the characteristicstill objects 601, and a train is the moving object 602. Subsequentimage processing methods will be described.

One of the image processing methods is a method in which predictionimage information is partially generated by the AI unit 104 with respectto only a portion determined as the moving object 602 in the imageinformation. Then, prediction image information based on the imageinformation about the still object 601 is not generated because the realimage of the still object 601 is less likely to be changed. Predictionimage information 610 generated by the above method is illustrated inFIG. 6B. For illustrative purposes, the moving object 602 in FIG. 6Aremains illustrated in FIG. 6B. In the prediction image information 610,a position of a moving object 612 is different from a position of themoving object 602, and positions of still objects 611 are not changedfrom the positions of the still objects 601 (not illustrated). Throughthe above-described processing, a load of the processing unit 102 can bereduced while displaying the image in real time.

Another image processing method is a method in which additional imageprocessing is further executed with respect to only a portion determinedas the moving object 602. In the additional image processing, resolutionof the portion determined as the moving object 602 is improved and/orrefined, whereas resolution of each of portions determined as the stillobjects 601 is lowered. The high-resolution moving object 612 and thelow-resolution still objects 611 are displayed on the display unit 103.Because processing is changed depending on the portion, a load of theprocessing unit 102 can be reduced. Further, because the moving object612 is displayed at high resolution and the still objects 611 aredisplayed at low resolution, a natural image close to an image seen bythe human eyes can be provided.

Yet another image processing method is a method in which predictionimage information is generated with respect to a portion of a stillobject in the periphery of the moving object 602 if there is time beforeoutputting a signal to the display unit 103 after the prediction imageinformation is generated with respect to the moving object 602. Withthis method, an image with higher precision can be displayed in realtime.

As described above, the processing unit 102 executes moving objectdetection on the image data received from the imaging unit 101 andchanges the processing method depending on the portion. With thisprocessing method, an image can be displayed with a high quality whilereducing a load of the processing unit 102.

In addition, a method other than a method using two or more pieces ofimage information is also provided as a method for detecting a stillobject and a moving object. For example, a moving object detection unitmay be provided as the detection unit 107. The moving object detectionunit may include a range-finding sensor. The number of the still objects601 and the moving objects 602 to be detected by the moving objectdetection unit is not limited.

Hereinafter, a fourth exemplary embodiment will be described. An imagingdisplay device of the present exemplary embodiment will be describedwith reference to FIGS. 7 and 8 . FIG. 7 is a schematic diagramillustrating an imaging display device 700 of the present exemplaryembodiment. The imaging display device 700 includes a line-of-sightdetection unit 108 in place of the detection unit 107 included in theimaging display device 400 illustrated in FIG. 4 . In FIG. 7 , theline-of-sight detection unit 108 is built into the imaging displaydevice 700. Alternatively, the line-of-sight detection unit 108 may bearranged externally.

In FIG. 7 , image information captured by the imaging unit 101 is outputto the processing unit 102. At the same time, line-of-sight informationacquired by the line-of-sight detection unit 108 is output to theprocessing unit 102. The processing unit 102 generates prediction imageinformation by the AI unit 104. Further, the processing unit 102generates prediction line-of-sight information by predicting movementand a position of the line-of-sight in the future based on theline-of-sight information. Then, based on the prediction line-of-sightinformation, the processing unit 102 executes additional imageprocessing for improving the resolution of a portion where aline-of-sight exists and lowering the resolution of a portion other thanthe portion where the line-of-sight exists, and generates finalprediction image information. While real time display is being executed,the display unit 103 can display the portion where the predictedline-of-sight exists at high resolution, and display the portion otherthan the portion where the predicted line-of-sight exists at lowresolution.

This operation will be described in detail with reference to FIG. 8 .FIG. 8 is a diagram illustrating the operation of the imaging displaydevice of the present exemplary embodiment. Similar to FIG. 5 , FIG. 8illustrates a scene in which persons are playing baseball. FIG. 8illustrates examples of three types of images captured at four differenttimes T⁻³, T⁻², T⁻¹, and T₀.

The real image refers to an image captured at each of the times T⁻³,T⁻², T⁻¹, and To. A batter hits and returns a ball at the time To. Thedescriptive image schematically illustrates a line-of-sight(line-of-sight area) on a real image. The line-of-sight is detected bythe line-of-sight detection unit 108. In the descriptive image, aline-of-sight is adjusted to a ball and is moved along with the movementof the ball. The image of the present exemplary embodiment is an imageobtained using the prediction line-of-sight information of the presentexemplary embodiment. This image is based on the prediction imageinformation on which image processing for improving the resolution ofthe portion where the line-of-sight exists and lowering the resolutionof the portion other than the portion where the line-of-sight exists isexecuted additionally. A portion of the ball is displayed at highresolution, and a portion of the pitcher is displayed at low resolution.By the above-described processing, a high quality image can be displayedin real time while reducing the load of the processing unit 102.

Although the imaging display device 700 includes the line-of-sightdetection unit 108 in place of the detection unit 107 illustrated inFIG. 4 , the imaging display device 700 may include both of thedetection unit 107 and the line-of-sight detection unit 108. Theconfiguration thereof can be changed as appropriate. Further, theline-of-sight detection unit 108 can employ an optional method such as amethod for detecting a position of the iris of the eye or a method usingcorneal reflection by emitting infrared light.

Hereinafter, a fifth exemplary embodiment will be described. In thepresent exemplary embodiment, a processing method to be executed whenthe line-of-sight detection unit 108 of the fourth exemplary embodimentdetects a plurality of line-of-sight areas will be described withreference to FIG. 9 . FIG. 9 is a diagram illustrating the operation ofthe imaging display device of the present exemplary embodiment. Similarto FIG. 8 , FIG. 9 is a diagram illustrating a scene in which personsare playing baseball. FIG. 9 illustrates three types of examples ofimages captured at three different times T⁻³, T⁻², and T⁻¹. In thepresent exemplary embodiment, prediction image information in which aplurality of line-of-sight areas is weighted and processed to resolutionbased on the weighting is generated. The display unit 103 displaysimages of portions corresponding to the line-of-sight areas atresolution based on the weighting, and images of portions other than theline-of-sight areas at low resolution.

The operation will be described in detail with reference to FIG. 9 . Areal image is an image captured at each of the times T⁻³, T⁻², T⁻¹. Eachof the real images includes a pitcher and a runner on first base, and abatter is about to hit and return the ball at the time T⁻¹.

The descriptive image of FIG. 9 schematically illustrates aline-of-sight area on a real image. A weighting value (%) is indicatedfor each of the line-of-sight areas. At the time T⁻³, a line-of-sight ofthe batter is adjusted to the ball thrown by the pitcher and to therunner. Herein, the line-of-sight area on the ball is weighted by 60%,and the line-of-sight area on the runner is weighted by 40%. At the timeT⁻², the line-of-sight of the batter is adjusted to the ball. Therefore,the line-of-sight areas on the ball, the runner, and the pitcher areweighted by 90%, 8%, and 2%, respectively. At the time Li, theline-of-sight of the batter is mostly adjusted to the ball. Theline-of-sight areas on the ball, the runner, and the pitcher areweighted by 98%, 1%, and 1%, respectively. This weighting is executed bythe processing unit 102, and a value of the weighting may be determinedbased on the movement of the line-of-sight detected by the line-of-sightdetection unit 108. Alternatively, the determination may be executed byanother AI unit 104.

The image of the present exemplary embodiment illustrated in FIG. 9 isan image using the prediction line-of-sight information according to thepresent exemplary embodiment. The processing unit 102 adjusts resolutionof a portion based on the weighting of the line-of-sight areaillustrated in the descriptive image, and generates prediction imageinformation. Each value illustrated in the image of the presentexemplary embodiment represents a ratio of resolution, and this is avalue when the maximum resolution is 100%. The portions other than theline-of-sight area can be displayed at the minimum resolution. With thisconfiguration, an image close to an image seen by the human eyes can bedisplayed, and a processing load of the processing unit 102 can also bereduced.

Hereinafter, a sixth exemplary embodiment will be described. An imagingdisplay device of the present exemplary embodiment can display an imageusing light (near-infrared light, infrared light, and ultraviolet light)other than visible light. For example, in the imaging display device 400illustrated in FIG. 4 , the imaging unit 101 includes a photoelectricconversion element that executes detecting a visible light area and aphotoelectric conversion element that executes detecting light of awavelength band falling outside the visible light area. For example, theimaging unit 101 includes at least two area sensors. On one of the twoarea sensors a photoelectric conversion element for visible light isarranged, and on the other area sensor, a photoelectric conversionelement for non-visible light is arranged. Alternatively, the imagingunit 101 includes one area sensor. The area sensor includes at least onephotoelectric conversion element for visible light and one photoelectricconversion element for non-visible light.

By the above-described imaging unit 101, an image signal of thenon-visible light area including a near infrared light area can beacquired in addition to the image information on the visible light area.Based on the above-described image information, the processing unit 102generates prediction image information on one visible light area. Withthis configuration, an image with improved and/or refined sensitivitycan be displayed even in a state where sensitivity is low in a visiblelight area. In other words, an image that cannot be seen by the humaneyes is also displayed in real time by the imaging display device of thepresent exemplary embodiment. The above-described imaging display deviceof the present exemplary embodiment is applicable to, for example, anight-vision device, a monitoring device, binoculars, a telescope, and amedical detection device.

Hereinafter, a seventh exemplary embodiment will be described. In theabove-described exemplary embodiment, prediction image information isgenerated by AI processing using a deep learning function. In thepresent exemplary embodiment, a trained model established throughmachine learning is used. Data is collected from specialists, and atrained model is established based on the collected data. Then, animaging display device in which prediction image information isgenerated based on the data collected from the specialists is applied toa non-specialist. For example, data is acquired from professionalathletes, and a trained model is established based on the acquired data.Then, prediction image information is generated by an AI unit using thistrained model. By using the imaging display device of the presentexemplary embodiment, a non-professional athlete can virtuallyexperience a line-of-sight or attentiveness of a professional athlete,so that the athletic skill of the non-professional athlete can beimproved and/or refined within a shorter period of time. The presentexemplary embodiment is also applicable to a field where inheritance ofspecialized skills of an expert is desired. For example, the presentexemplary embodiment is applicable to occupations in various fields thatrequire specialized skills, such as a pilot, a doctor, a traditionalcraftsman, and a security/safety service.

Hereinafter, an eighth exemplary embodiment will be described. Anapplication example of the imaging display device according to each ofthe above-described exemplary embodiments when applied to a wearabledevice will be described with reference to FIGS. 10A to 10E. The imagingdisplay device can be applied to a wearable device, such as asmart-glass, a head-mounted display (HMD), or a smart contact lens.

FIG. 10A is a schematic diagram illustrating a smart-glass 1000. Thesmart-glass 1000 is also called as an eyeglass-type imaging displaydevice or eyeglasses. The smart-glass 1000 includes an eyeglass frameand an imaging display device according to the above-described exemplaryembodiments. Specifically, the smart-glass 1000 includes at least twoimaging units 1001, a processing unit 1002, and a display unit 1003. Thetwo imaging units 1001 are arranged on side faces of the eyeglass frame,and the processing unit 1002 is housed within a temple of theeyeglasses. The display unit 1003 is arranged on an optional positiondepending on a display form, and may be included in a lens 1011. In anyof the cases, the display unit 1003 displays an image on the lens 1011.The processing unit 1002 may include an AI unit. The smart-glass 1000may include an external interface, so that the processing unit 1002 mayexchange data with an external AI unit.

The smart-glass 1000 in FIG. 10A may include two respective imagingdisplay devices for the right and left eyes. In this case, timings forcapturing and displaying images can be optionally set in the respectiveimaging display devices for the right and left eyes. Specifically,images can be captured simultaneously and displayed at different times,or images can be captured at different times and displayedsimultaneously.

FIG. 10B is a schematic diagram illustrating a smart contact lens 1020.The smart contact lens 1020 is also called as a contact lens-typeimaging display device or a contact lens. The smart contact lens 1020includes one imaging display device 1021 and one control device 1022.The control device 1022 functions as a power source unit for supplyingpower to the imaging display device 1021. The control device 1022includes an AI unit and supports a processing unit of the imagingdisplay device 1021. In addition, the AI unit may be arranged on aterminal different from the smart contact lens 1020. It is desirablethat an optical system for collecting light to the imaging displaydevice 1021 be arranged on the smart contact lens 1020. The power sourceunit includes an interface for connecting the power source to anexternal portion. The power source unit may be connected to the externalportion and charged via wired connection or wireless connection.

A transparent material is used as base materials of the lens 1011 inFIG. 10A and the smart contact lens 1020 in FIG. 10B, and a display unitof the imaging display device projects a display image on a transparentlens portion. At this time, an image based on the prediction imageinformation at a future time later than a display time as illustrated inFIG. 2B can be displayed, so that the user can see both of the realimage and a prediction image in the future. Because image information atthe time slightly later than the current time can be displayed in realtime, for example, the user playing outfield in baseball can take adefensive position in advance by moving in a direction of a ball hit bythe batter. At this time, the user can realize a higher level ofathletic performance than the actual athletic performance level of theuser because the user can see both of the real image and the predictionimage. Further, the imaging display device can freely adjust the timingfor displaying the acquired image information. With this configuration,the operation suitable for the user can be selected.

As illustrated in FIG. 10A, the imaging unit 1001 and the display unit1003 may be arranged at different positions, or may be laminated andarranged on the line-of-sight. FIG. 10C is a cross-sectional schematicdiagram illustrating the imaging unit 1001 and the display unit 1003.FIG. 10D is a planar schematic diagram of the imaging unit 1001 and thedisplay unit 1003 viewed from a side of the imaging unit 1001. FIG. 10Eis a planar schematic diagram of the imaging unit 1001 and the displayunit 1003 viewed from a side of the display unit 1003. FIG. 10Dillustrates a centroid of an imaging area 1031 where pixels of theimaging unit 1001 are arrayed. FIG. 10E illustrates a centroid of adisplay area 1033 where pixels of the display unit 1003 are arrayed. Asillustrated in FIG. 10C, it is desirable that the imaging unit 1001 andthe display unit 1003 be arranged in such a state that a line segment Apasses through the two centroids. With this configuration, it ispossible to reduce variation in the wearable devices, which is caused bydifference in position between the captured image information and adisplay image.

Hereinafter, a ninth exemplary embodiment will be described. In thepresent exemplary embodiment, an imaging display system will bedescribed. FIG. 11 is a schematic diagram illustrating an imagingdisplay system 1100. The imaging display system 1100 of the presentexemplary embodiment includes a plurality of imaging display devices1101 and at least one control device 1102. The imaging display device1101 may be an imaging display device described in any one of theexemplary embodiments, e.g., the imaging display device 100 illustratedin FIG. 1A.

The plurality of imaging display devices 1101 can receive and transmitsignals from and to the control device 1102. Each of the imaging displaydevices 1101 and the control device 1102 includes an external interfaceunit for executing wired or wireless communication. The control device1102 receives signals from the plurality of imaging display devices 1101and outputs signals for controlling the plurality of imaging displaydevices 1101. The control device 1102 may include a part of thefunctions of the processing units 102 of the imaging display devices1101. The imaging display system 1100 may further include a data storageunit, a control unit, and a processing unit. For example, the imagingdisplay system 1100 may include the processing apparatus 105 or 106illustrated in FIG. 1D. In this case, the control device 1102 cancommunicate with the processing apparatus 105 or 106.

For example, the imaging display system 1100 of the present exemplaryembodiment can display an image on a single display unit 103 by usingpieces of image information acquired from a plurality of respectiveimaging units 101 included in a plurality of imaging display devices100. Specifically, for example, when a plurality of users uses therespective imaging display devices 100, pieces of image information isacquired from the plurality of imaging display devices 100, and imagescan be displayed on another imaging display device 100 used by a userdifferent from the plurality of users in real time. For example,line-of-sight images of professional athletes playing in the sameathletic filed can be concurrently seen by at least one spectator inreal time.

Further, images may be displayed on a plurality of display units 103using the image information acquired from a single imaging unit 101.Specifically, image information acquired from one imaging display device100 can be displayed on a plurality of imaging display devices 100different from the one from which the image information is acquired inreal time. For example, a line-of-sight image of one professionalathlete can be concurrently viewed by a plurality of spectators in realtime.

As described above, by virtually experiencing the line-of-sight of theprofessional athlete, the spectator can view an image with a sense ofpresence which makes the spectator feel as if the spectator himselfexists in the athletic field.

Further, this system allows respective display apparatuses to executeimaging operation or display operation at different intervals. Further,in the system, image information and various types of informationacquired by a plurality of imaging display devices can be shared andused for creating pieces of prediction image information on theplurality of imaging display devices.

Hereinafter, a tenth exemplary embodiment will be described withreference to FIG. 12 . FIG. 12 is a schematic diagram illustrating animaging display device 800 of the present exemplary embodiment. Theimaging display device 800 illustrated in FIG. 12 is similar to theimaging display device 130 in FIG. 1C. The same reference numerals areapplied to the same elements, and descriptions thereof will be omitted.The imaging display device 800 includes an imaging unit 101, aprocessing unit 102, a display unit 103, and a processing apparatus 105.The imaging display device 800 further includes a recording unit 109 forrecording image information. In the imaging display device 800, each ofthe processing unit 102 and the processing apparatus 105 includes an AIunit 104. Although the imaging display device 800 includes the recordingunit 109 in FIG. 12 , the recording unit 109 may be included in theprocessing unit 102. A position of the recording unit 109 can be set asappropriate. Information to be input to the recording unit 109 is, forexample, image information from the imaging unit 101, i.e., informationbefore being converted to prediction image information. By recording theabove-described image information, the recording unit 109 can alsoacquire an image other than a prediction image.

Operation of the imaging display device 800 according to the presentexemplary embodiment will be described. The image information from theimaging unit 101 is input to both of the recording unit 109 and theprocessing unit 102 that generates prediction image information.Operation to be executed after the image information is input to theprocessing unit 102 is similar to the operation described in the firstexemplary embodiment. The image information input to the recording unit109 is not converted to a prediction image, and directly recorded as theimage information acquired by the imaging unit 101. With thisconfiguration, while a temporal difference between the actual phenomenonand a display image is reduced by using the prediction image informationgenerated by the processing unit 102, image information acquired by theimaging unit 101 can be recorded as it is.

Hereinafter, an eleventh exemplary embodiment will be described. In thepresent exemplary embodiment, a method in which enlargement processingis further executed by the processing unit 102 and the AI unit 104 ofthe first exemplary embodiment will be described. FIG. 13 is a diagramillustrating the operation of the imaging display device of the presentexemplary embodiment. Similar to FIG. 6A, external information in FIG.13 represents a real image, i.e., image information acquired by theimaging unit 101.

An example 1 is a comparison example. An enlarged image of the example 1illustrates partial image information acquired by executing enlargementprocessing on one portion as a specific area of the image informationacquired by the imaging unit 101. Herein, the display unit 103 displaysan image based on the partial image information. As illustrated in theexample 1, generally, resolution of the partial image information islowered when the enlargement processing is executed thereon.

An example 2 is an example according to the present exemplaryembodiment. The enlarged image of the example 1 illustrates partialimage information acquired by executing enlargement processing on oneportion as a specific area of the image information acquired by theimaging unit 101. Herein, the display unit 103 displays an image basedon the partial image information. In the example 2, the processing unit102 executes enlargement processing on the one portion, and furtherexecutes resolution improvement processing thereon. Resolutionimprovement processing is processing for improving the resolution. Forexample, the resolution improvement processing can be executed on thepartial pixel information by executing inter-pixel complementaryprocessing using a plurality of pieces of image information. Further,resolution improvement processing is executed by complementaryprocessing using a plurality of pieces of image information orcomplementary processing for estimating a shape by detecting a contourof image information.

In the imaging display device of the present exemplary embodiment,enlargement processing of a part of the image based on the imageinformation can be executed by the processing unit 102 and the AI unit104. Thus, the image based on the enlarged partial image information canbe displayed on the display unit 103. The imaging display device of thepresent exemplary embodiment can execute resolution improvementprocessing on the partial image information. Further, the image based onthe enlarged partial image information described in the presentexemplary embodiment can be realized in concurrence with the functionfor reducing a temporal difference between the actual phenomenon and thedisplay image as described in the other exemplary embodiments.

Although the exemplary embodiments have been described by takingbaseball as an example, the present disclosure is not limited thereto.The imaging display device of the present disclosure can reduce atemporal difference between the actual phenomenon and the display image,so that the user can use a device without having a sense of discomfort.As described above, according to the aspect of the present disclosure,an imaging display device that executes reducing a difference betweenthe actual phenomenon and the display image can be acquired.

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may include one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random access memory (RAM), a read-only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the disclosure is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. A device comprising: an imaging unit including aplurality of photoelectric conversion elements; a processing unitconfigured to process a signal transmitted from the imaging unit, andincluding an artificial intelligence (AI) unit; and a display unitconfigured to display an image based on the signal transmitted from theprocessing unit, wherein the processing unit is configured to processfirst image information at a first time from the imaging unit, andconfigured to generate first prediction image information with respectto a second time later than the first time based on the first imageinformation.